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Abstract—In addition to autonomous car operation, in many 
cases it is desirable to let a human drive the vehicle remotely. 
To make remote operation possible, it is very critical to have 
a low and predictable latency to transmit video from the car 
cameras and receive control commands back. In this paper, 
we analyze the problem and present a communication and 
video streaming system that addresses the latency challenges and 
enables teleoperation of a real car over commercially available 
LTE network; demonstrating sub-50ms roundtrip latencies for 
720p, 60FPS video, with average PSNR 36db. 


I. INTRODUCTION 


EMOTE driving is a term used to describe the teleop- 

eration of a car where the driver is physically located 
far away from the car, but can see video from a windscreen 
mounted camera, turn the steering wheel, and press the pedals 
to behave like a real driver, with steering, braking and accel- 
eration commands sent to the car as shown in Figure 1. 

Today remote driving is considered as a companion to 
almost all autonomous driving system to increase overall car 
operation reliability and safety. For example, in the case of 
an unexpected event—like a road work enforced lane change, 
flood, fire, emergency or similar—when the autonomous driv- 
ing system does not know how to react, it requests a remote 
driver to connect to the car, evaluate the situation, and continue 
driving; instead of stopping and making an autonomously 
driven car a problem to other traffic parties [1]. 

To enable the remote driving, many technical problems 
should be solved: a car needs to have Drive-By-Wire (DbW) 
capability that assumes electronic, not mechanical control of 
steering, braking, and other systems; a system to transmit 
video and other measurement data from the car to the remote 
operator; and a similar system to send and receive commands 
from the remote operator. 

Latency is a key factor in DbW control, video compression, 
and data transmission and reception. However, in the literature 
the authors did not find exact numerical parameters (like av- 
erage and variance) that are required to say that some number 
is a maximum, i1.e., if a parameter of the system is greater 
than some value, then the usage is not possible. We decided 
to build a prototype to answer the question with practical tests 
and measurements similar to [2] where communication latency 
acceptability was studied with the real experiment of robot 
teleoperation. 

The end-to-end latency of any teleoperation, including re- 
mote driving, is more than just the network packet roundtrip 
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Fig. 1. Remote driving framework. 


time. The true end-to-end operating latency is a sum of all 
individual framework component latencies, including low- 
level processing and buffering. There is no single hotspot that 
can be optimized to reduce the overall latency dramatically as 
there are many components that are either out of an engineer’s 
control or very hard to modify. For example, displaying 
framebuffer is a source of latency, just as any other block- 
based processing element in the system, since it waits for the 
whole block to be received before continuing. Table I shows 
typical latency sources in the system in the same order as 
the video is transmitted from the source to the remote driver 
camera capture, framebuffer and display refresh are connected 
to frame rate, 60fps in this case. Video compression and de- 
compression, LTE and internet communication links introduce 
variable latency and are as indicated in Table I. 


TABLE I 
SOURCES OF LATENCY IN REMOTE DRIVING 


Source Latency Cumulative Latency 
[ms] [ms] 
Camera capture, 60fps 17 17 
Video compression 17 334120 6 ere is 
LTE+Internet one way, | 10 ...300 44 ...437 
1.5K packet 
Video decompression 17 ...32 61 ...469 
Framebuffer 17 78 ...486 
DVI, LCD refresh 10 88 ...496 


In this paper, we present a way to reduce the end-to- 
end latency significantly, so even with existing LTE mobile 
networks it would be possible to enable latency critical work- 
loads like remote teleoperation of a real car without any 
special agreement from cellular carriers, such as free traffic or 
prioritization of data. We present a remote driving framework 
in which we tackled two main latency sources: 


1) Network latency and buffering; 

2) Video encode and buffering. 
The paper is organized as follows: in Section II, we review 
the theory of the network latency reduction with forward error 
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Fig. 2. An example of theoretical latency gain of network coding 


correction coding [3] and present hypothetical estimations of 
gains. However, to enable the theory, many implementation 
considerations are discussed in Section III. In Section IV, we 
describe a video compression scheme and its practical imple- 
mentation for remote car operation. Section V summarizes the 
results. 


II. FORWARD ERROR CORRECTION APPLICATION AT THE 
NETWORK LAYER TO REDUCE TRANSMISSION LATENCY 


To send a large data block (like a video slice), the applica- 
tion has to split data into K smaller packets in accordance to 
the physical, MAC and higher layer protocols. Then at the 
receiver side, the whole data block can be processed only 
after all of its contents are received. So, the transmission 
latency is determined by the latest packet of the original block 
that arrived. Network layer coding is a well-known theory to 
reduce delay in packet switching networks [3] by introducing 
redundancy: original K packets are considered as symbols 
of some erasure correction code (ECC) and encoded into 
N > kK symbols (packets), then all N packers are sent to 
the network. Usually, in accordance to the properties of the 
ECC, the decoder can reconstruct the codeword by any K 
out of N packets, so the latency of the whole block will be 
determined by the latest out of K first received packets. (k- 
order statistic). To demonstrate the effect of the network layer 
coding in terms of reconstructed block latency distribution, 
consider a theoretical example below. 

Assume we transmit data blocks that will be split into 10 
packets, and the packet delay time are independent Gaussian 
random numbers with mean = 20 and stddev = 5. Original 
reconstructed block latency can be drawn as a cumulative 
distribution function (CDF) of maximum of 6 samples. If we 
encode 6 packets into 8 and assume that the network load did 
not change, so the packet delay time has the same distribution, 
we can plot a 6-order statistic for 8 samples, and it will 
represent a theoretical latency of the reconstructed blocks with 
the network layer coding (code rate = 3/4), see Figure 2. At 
the level of probability = 0.95 one can see the difference of 
6. So, if random numbers represent milliseconds of a packet 
delay, in the given example the network coding can provide 
6ms latency improvement. 

The numbers look so great so many companies wanted 
to use network layer coding for latency critical applications 
like voice/video conferencing, control systems and other and- 
user applications over the Internet. But in practice, end-user 
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Fig. 3. Latency of a constant rate stream measured at the receiver and 
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applications are different than the ideal model because of the 
following: 

1. Erasure correction codes used in applications today are 
not optimal. In 1985, when Kabatiansky and Krouk first 
published their paper that coding reduces latency, the compu- 
tational complexity of encoding/decoding was a big concern, 
so many other researchers proposed code constructions like 
LT codes, Fountain and Raptor codes [4] that provided linear 
decoding complexity. They are very implementation friendly, 
but unlike Reed-Solomon codes that are optimal in terms of 
capacity, Raptor codes are not. 

2. Data buffers in modems and other network interfaces. 
When an application opens an IP socket and sends a datagram, 
it does not mean that the packet is actually sent to the physical 
layer of the network. There are many buffers at the operating 
system level also as at the level of the device, so totally 
some number of megabytes can be buffered before physical 
transmission. Figure 3 presents an example of a packet latency 
measured at the receiver for MBps constant transmission rate 
where one can see that at some time latency values grow to 
200ms meaning that the packets were just buffered somewhere 
in the transmission pipeline when the channel rate was low, 
and when the network conditions changed, packets started to 
arrive. 

3. Packet delay times are not independent. The Internet 
itself is a multi-hop packet switching network at the level of 
switches and routers, but when it comes to the last mile, a 
typical user has only one line of connection like shown in 
Figure 4, and this last mile makes packet delay times very 
correlated: if the last mile packet is delayed, the probability 
that the next packet will be also delayed is very high as there 
is only one physical link, and packets are sent one-by-one into 
it. In Figure 3 the graph at the bottom shows autocorrelation 
of 500 best packet latencies, and it is very high. 


Ill. PRACTICAL ASPECTS OF LOW LATENCY 
TRANSMISSION FOR COMMERCIALLY AVAILABLE LTE 


In this section we describe practical methods of solving 
three problems mentioned above to enable latency reduction 


with network coding. 


A. High-throughput implementation of optimal erasure cor- 
rection codes 


At the time of Raptor codes appearance, Reed-Solomon 
based erasure control codes were considered as not im- 
plementable to provide data rates at the order of number 
of megabytes per second and above. But since that time, 
computer hardware and instruction set architectures has made 
significant progress, and today it is not a problem to reach hun- 
dreds of megabytes per second with Cauchy Reed-Solomon 
codes (CRS) using SSE2 [5]: 

The target data rates for LTE modems we planned to use is 
in the order of 1-2MBps, we have chosen CRS codes as they 
always provide required data rates and are optimal from the 
redundancy point of view. Table II lists the CRS performance 
for a packet size of 1296 bytes under various redundancy 
levels. 


TABLE II 
CRS PERFORMANCE FOR PACKET LENGTH 1296 BYTES 


Code message length K Encoder Decoder 
and number of [Mbps] [Mbps] 
redundant symbols M 

K=100, M=1 23314 

K=100, M=10 1171 

K=100, M=20 579 

K=100, M=30 347 

K=100, M=40 274 

K=100, M=50 218 


B. Instant rate control and scheduling make possible to avoid 
buffering 

To avoid packet buffering at the transmitter side, we used 
a feedback-based rate control that measures the data rate 
at the receiver, makes a decision about the best rate for 
the current channel state and sends this information to the 
transmitter. Because we deal with video, it is always possible 
to increase or decrease compression ratio to match the channel 
rate requirements set by the rate control. There are 2 states of 
the channel rate control algorithm: 


1) Measurements and decision making. In this mode, if the 
measured data rate is less than the target, the target rate 
is set to the measured minus some delta to empty buffers 
if they have extra packets. 

2) Rate probation. The algorithm enters probation state 
if there were no rate decreases for some number of 
measurements. In probation state, the algorithm requests 
temporary rate increase by some delta followed by the 
rate decrease by the same delta, and then measures in- 
stant data rates at the receiver. If measurements show the 
same increased and decreased rate, then the algorithm 
sends a permanent rate increase command, otherwise the 
rate remains unchanged. 

As we will have multiple links, we need a scheduler mecha- 
nism to distribute data packets over multiple transmitters. For 
each link, the scheduler intoduces a parameter we refer to as 
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Fig. 4. A typical end user connected to the Internet with a single link. 


Provider 1 


Modem/NIC Sddctiser 


LJ 


Fig. 5. Configuration with multiple modems from different carriers. 
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an anticipated transmission end time, and each time a new 
packet is given to the scueduler, the packet is assigned to the 
link with the closest transmission end time. At the beginning, 
the anticipated transmission end time parameter is set to the 
current time. Then the scheduler counts the amount of data 
to be sent over the link, also takes measurements from the 
rate control to determine the expected time of transmission 
by simply dividing the size by the rate and increases the 
anticipated transmission end time by the expected time of 
transmission. 


C. Last mile connection with multiple LTE modems 


We used an extensive way to solve the last mile packet delay 
time correlation problem: instead of just one LTE modem, we 
used 4 modems of the same type with 4 SIM cards (2 SIMs 
from one carrier and 2 SIMs from another carrier) as shown 
in Figure 5. Experiments show that even using 2 modems the 
latency autocorrelation function decreases significantly (see 
Figure 6), and if the number of modems becomes 4, there 
is no correlation in packet latency at all. 

After implementation of the rate control, scheduling and 
network coding described above, we have a networking mod- 
ule depicted in Figure 7. This module was using 2 modems 
(SIMs were from 2 different carriers) using the following 
scenario: we tested average data rates that we can have with 
the rate control for each link and measured packet latency. 
These rates were approximately 1 and 0.75Mbps. Then we 
generated a stream of data blocks with the rate of 1MBps 
and encoded it with the code at rate 4/7 so the resulting data 
rate was approx. 1.75MBps that is a sum of the individual 
modem rates. Then we transmitted encoded packets over the 
same network at the same place, used FEC decoding at 
the receiver to reconstruct blocks right after we had enough 
packets and measured the latency. The latency CDF is shown 
in Figure 8. Curves “modem (1)” and “modem (2)” represent 
a hypothetic reconstructed latency of the same block without 
coding using the latency statistics measured at the beginning 
of the experiment. At the level of probability = 0.95 we had 
latency gain of 8.5ms. 
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Fig. 6. Latency and autocorrelation of packet latency of a constant rate stream 
transmitted over 2 LTE modems. 
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Fig. 7. Networking module scheme. 


IV. SLICE BASED VIDEO COMPRESSION SYSTEM 


A traditional 60fps camera outputs video at the frame 
granularity, i.e. every ee of a second it outputs a data block 
corresponding to the whole frame. There is no other way to 
make the latency lower but to reduce the granularity. At the 
time of making the prototype, we did not find any cameras in 
the market that were able to output data at a granularity lower 
than one frame, so we developed a dedicated system with the 
following requirements: 


1) Read a slice (a horizontal part of the frame) from the 
image sensor and pass it to the compression. We have 
chosen to read data with the granularity of , of the frame 
at 60fps, so the read latency = a = 0.002(7)sec.; 

2) Get the current available rate from the networking rate 
control and scheduler module and compress image slice 
to be transmitted in accordance with the compression 


ratio derived from the desired data rate. 


A. H.264 intra refresh compression mode for streaming 


We have chosen H.264 [6] as a video compression option 
as it provides high compression ratios with a various number 
of features to recover from the packet loss recovery at the 
time of transmission, also there were many available libraries 
to decode H.264 streams in software. By streaming mode, we 
assume that at the receiver side the video frame needs to be 
displayed at a certain time no matter whether the decoder has 
all frame slices available or not. In other terms, no buffering, 
no waiting for the slices that are either delayed or lost in 
transmission. As we wanted to minimize the video display 
latency, we chose the display time close to transmission 


reconstructed slice latency CDF 


F(x) 


modem(1), actual 0.75MBps 
modem(2) actual IMBps 
joint coded R=4/7 


Fig. 8. Reconstructed slice latency for 2 independent modems and joint 
network coded at the same data rate. 
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Fig. 9. Sweeping I-macroblock pattern in a P-slice. 


reconstruction time at 95% probability, so we had a non-zero 
probability of slice loss at the input of the video decompressor 
that can be roughly approximated in a following way: one slice 
out of 20 is lost. In this case, lossy reconstruction features of 
the H.264 standard became very important. To operate in the 
case of slice loss, we used an option in H.264 called “intra 
refresh’, when intra-predicted macroblocks are systematically 
placed in the inter-predicted slices instead of sending entire 
intra frames, see Figure 9. 

In our scheme, we have chosen columns of I-macroblocks 
spaced every 16 macroblocks as the granularity of intra 
refresh; at any slice N, if the I-macroblock columns appear 
at index 0,16,32,... the subsequent slice (NV + 1) has I- 
macroblock columns at indices 1,17,33,.... At slice N + 16, 
the I-macroblock columns restart from indices same as that of 
slice N.. With this mechanism, even with some dropped frames 
in between, recovery of the image happens within 16 frames. 
In a camera with 60fps frame capture rate, the recovery time 
will be 0.26s which is same as the average reaction time of a 
human to visual stimulus. The recovery time can be adjusted 
based on the repetition rate of the I-macroblocks at the cost 
of slightly higher data rate requirement. I-macroblocks with 
a periodicity of 8 reduces the recovery time by 2x (to 0.13s) 
and so on. A recovery rate of < 0.26s suits perfectly well to 
requirements such that any dropped frames can be recovered 
without the remote driver/operator noticing any change in the 
video being transmitted. Besides just visual evaluation, we 
have studied the effect of intra-refresh in the case of packet 
loss comparing 3 modes of the H.264 encoder that we have 
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Fig. 10. Rate-distortion graph for Intra, Inter and Intra-refresh. 


implemented: intra-only mode (all frames are encoded as intra, 
so they don’t require reference frames to be decompressed), 
inter-only mode (only the first frame in the stream is encoded 
as intra, and all others are encoded as inter referring always to 
the previous frame) and intra-refresh mode (described above). 
We present PSNR/rate graph for image sequences of 16 slices 
captured with the sensor mounted on the car windscreen and 
is presented in Figure 10. Here we intentionally dropped one 
compressed slice, and then decompressed the rest 15 with 
FFMPEG [7] software and measured average PSNR over the 
decompressed slices assuming that the first missing slice was 
directly duplicated from the slice previous to the dropped one. 
As one can see from the graph, the encoder in intra-refresh 
mode outperforms others at the most important PSNR range 
(about 36 — 37dB), so we have proven that intra-refresh is the 
best option. 


B. Real time quantizer update 


H.264 standard provides developers with a lot of options 
affecting compression ratio, but for simplicity, in our system 
we used only quantization parameters update for I and P 
macroblocks to change the size of the compressed slices. As 
our target was low latency, we implemented a simple lookup- 
table based quantizer selection procedure based on statistical 
data. In the video compression unit, we implemented cyclic 
sequential increase and reset of the quantization parameters, 
then a sensor mounted to the windscreen of the real car 
was driven on the road for 1 hour to collect statistics of 
compressed slice length as a function of I and P macroblock 
quantizers. The compression was done using intra-refresh 
option presented in the previous subsection. Then we averaged 
the statistics collected and built a look-up table where for each 
desired slice size had a corresponding pair of quantizers (for 
I and P macroblocks) to be used. A graphical representation 
of the table depicting the QP values for desired compressed 
slice sizes is presented in Figure 11. 
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Fig. 11. Graphical representation of the lookup table for quantizer selection 
as a function of a desired compressed slice size. 
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Fig. 12. Video compression architecture. 


C. Tightly coupled image sensor and ping-pong video encoder 
architecture 


H.264 encoder requires the whole slice to be available in its 
memory before starting compression because of predictions. 
Even in the case of reading data from the sensor directly, data 
needs to be buffered. Another important feature of the H.264 
compression is that its entropy encoding part operation time 
is data dependent, i.e. it is not possible to predict the exact 
number of operations that CABAC or CAVLC encoder will 
require in advance. In order not to waste the sensor buffering 
time, and in order to have a guard time interval for the entropy 
encoder to complete all its operations, instead of a single 
H.264 encoder, we decided to use 2 independent instances of 
the encoders. The architecture of using 2 encoders is shown 
in Figure 12; after reading image data from the sensor, it is 
converted to YUV420 format on the fly and stored in the local 
memory (slice buffer), and the first instance of the encoder 
starts to process the buffered video slice. At the same time, 
when the first encoder is processing the slice, sensor data 
reading and YUV420 conversion is continues to prepare the 
next slice writing it into which is eventually processed by the 
second instance of the H.264 encoder. The sensor continuously 
reads the image data and writes the slices into alternating 
buffers which are in turn processed by the encoders in a ping 
pong fashion. As a result, such a ping-pong scheme and 2 
instances of H.264 encoder doubles the slice reading time to 
complete all operations. In the case of 6 slices per frame, 
each encoder has °** = 0.00(5) seconds, that was our time 
boundary for the hardware. The overall latency of the sensor- 
compression pair was approx. 8.3ms. 

Figure 13 provides an approximate timing diagram of buffer 
usage and H.264 encoder processing for two scenarios: (1) 
single encoder instance; (2) 2 instances of H.264 in a ping- 
pong scheme as described above. We will show that the ping- 
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Fig. 13. Timing diagram of conventional vs. ping-pong mechanism. 


pong scheme is optimal from buffering and latency point of 
view. The top half of the figure shows the timing when a 
single encoder instance is used. The pixels from the video slice 
are buffered into slice buffers. The H.264 encoder can start 
only after the availability of slice 0 data. Since the processing 
exceeds the time interval of the next incoming slice, the new 
slice is buffered into buffer #1. As Slice 0 is still being 
processed, slice 2 data has to be buffered into buffer #3. 
The encoder sequentially processes each slice while the buffer 
requirement is at 3 times the size of one video slice. 

Now by doubling the H.264 encoders, we see that (from 
second half of Figure 13), there is a buffer associated with 
each encoder and they process either the even or odd numbered 
slices. Thus, the buffer requirement is reduced compared to 
when using a single encoder instance and at the same time 
halving the latency. Hence the ping-pong scheme improves 
the latency contributing to the overall reduction in latency of 
the entire system. 

The compression scheme is connected to the networking 
module: after receiving the desired rate, quantization parame- 
ters are chosen using the look-up table described in the previ- 
ous subsection and used in the compression. The compressed 
slices are sent to the networking module for transmission. 


V. SUMMARY 


Combining everything described above, we built a working 
prototype of the low latency video transmission system and 
demonstrated the remote car operation over commercially 
available LTE. 

With the prototype it was possible to achieve the following 
parameters: 46ms average network roundtrip time and 90ms 
average one-way video end-to-end latency measured with a 
standalone 60fps camera. The average network roundtrip time 
was measured directly by time stamps, and video end-to- 
end video latency was measured as the amount of time from 
any event that happens in front of the camera to the event 
displayed by a remote screen, see Figure 14. The measurement 
was performed by counting the number of frames in the 
video between 2 events. As the camera frame rate was 60fps, 
the granularity of the measurement was 16.67ms. With the 
parameters obtained, we were able to operate a real car 
remotely demonstrated by precise steering between the cones 
at the company’s test field as shown in Figures see 16 and 17. 

To achieve these latency numbers, we used two most 
impactful features: 


1. At the network layer, we introduced multiple links by 
use of more than 1 LTE modem. It was done not to increase 
the data rate, but to decorrelate the packet delays, so in 
combination with the rate control and scheduling, the principle 
of network layer coding made possible to lower the latency. 
As increasing the number of modems increases the cost of 
the system, we don’t recommend to make latency reduction 
schemes using more than 4 modems. While having 2 modems 
is the must, adding the 3rd modem gives very small latency 
gain in average, latency gains of adding extra modems after 
Ath would become marginal, as one can see in Figure 15, 
the average latency is changed by just Ims. It is important 
to note that adding of extra links changes the variance of the 
reconstructed slice delay: the more links we have, the lower 
is the variance. So, as the 4th modem does not change the 
average latency, but it makes the latency closer to constant. 

2. At the application layer, we introduced slice-by-slice 
video processing by splitting the whole frame into smaller 
parts and processing them independently, also changing com- 
pression parameters on the fly. To make it happen, we had 
to build a proprietary hardware as there were no components 
available in the market. The hardware implementation option 
was chosen not only because of the real time operation 
requirements as video compression by itself can be efficiently 
implemented in software using latest SSE instructions, but 
because of the high data rates of the video at 60fps. An 
HD 60fps sensor would generate an uncompressed stream at 
3GBps rate, so even 1Gbps ethernet that PCs have today is not 
enough to transmit the data. If compression used, the rate is 
lowered, but the latency grows, because the video stream needs 
to be transcoded in accordance with the required data rate at 
the network layer and has to be encoded using intra-refresh for 
streaming, but cameras available in the market use frame-by- 
frame compression that is another source of the latency and 
don’t generate streams within intra-refresh. Using hardware, 
we were able to read data from the sensor slice-by-slice and 
compress it providing data packets to the networking module 
for transmission directly. 

However, besides these two features we believe are most 
impactful, there is still room for the end-to-end latency im- 
provements. For example, frame buffers at the display side. 
The same way we presented with tightly coupling of the sensor 
and the compression can be used at the display side to let data 
remain in a form of small blocks at the granularity lower than 
one frame up to the very last point of the system that is a 
display matrix, so display manufacturers may consider this 
option for their new products in the future. 
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APPENDIX 


The final implementation of the networking module de- 
picted in Figure 7 that was used in remote driving is released 
as an open source Intel®Low Latency Multi Link Network 


Fig. 14. Video end-to-end latency measurement. 
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Fig. 15. Reconstructed slice latency distribution as a function of number of 
modems used. 


Coding Library under Apache License 2.0, so we welcome 
researchers to access it at GitHub [8]. 
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Fig. 16. Remote driving test setup inside the car. 


Fig. 17. Remote driving test field. 


